In the present analysis we will study the interest of the AI community on a AI benchmarks. We focus on “interest” rather than “progress” for AI benchmarks as this is something we can compute using some proxies. In this particular case, we use the number of (normalised) hits (number of documents) obtained from AItopics per benchmark over the last decade (2008-2019). Note that the results from 2019 are incomplete.
The benchmarks in the present analysis rely on our own previous analysis and annotation of papers, as well as on open resources such as Papers With Code, including data from from several repositories (e.g, EFF, NLP-progress, SQuAD, RedditSota, etc.).
In this two plots we show how the mean interest per AI benchmark has varied among different periods.
Note that we can perform exactly the same analysis focusing on different (ranges of) years and obtaining the same graph but the width of the edges may vary (a little bit).
(Network are interactive!)
plotVis(prepareVis(dataA)[[1]], categories = data$category)
plotVis(prepareVis(dataB)[[1]], categories = data$category)
plotVis(prepareVis(dataC)[[1]], categories = data$category)
plotVis(prepareVis(dataD)[[1]], categories = data$category)
cogAbs = c("MP", "SI", "VP", "AP", "AS", "PA", "CE", "CO", "EC", "NV", "CL", "QL", "MS", "MC")
barplot(colSums(select(data, cogAbs)))
periods <- list(2008:2010, 2011:2013, 2014:2016, 2017:2019)
all <- plotProgressPeriods(data, periods)
## [1] 2008 2009 2010
## Warning in if (interestYears == "All") {: the condition has length > 1 and
## only the first element will be used
## [1] 2011 2012 2013
## Warning in if (interestYears == "All") {: the condition has length > 1 and
## only the first element will be used
## [1] 2014 2015 2016
## Warning in if (interestYears == "All") {: the condition has length > 1 and
## only the first element will be used
## [1] 2017 2018 2019
## Warning in if (interestYears == "All") {: the condition has length > 1 and
## only the first element will be used
all.s <- summarise(group_by(all, period, variable), mean = mean(value))
ggplotly(ggplot(all.s, aes(variable,mean, fill = period)) + geom_bar(stat = "identity",position = "dodge") + xlab("") + ylab("Mean Interest") +
scale_fill_brewer(palette = "Paired") + theme_minimal())
periods <- 2008:2019
all <- plotProgressPeriods(data, periods)
## [1] 2008
## [1] 2009
## [1] 2010
## [1] 2011
## [1] 2012
## [1] 2013
## [1] 2014
## [1] 2015
## [1] 2016
## [1] 2017
## [1] 2018
## [1] 2019
all.s <- summarise(group_by(all, period, variable), mean = mean(value))
ggplotly(ggplot(all.s, aes(variable,mean, fill = period)) + geom_bar(stat = "identity",position = "dodge") + xlab("") + ylab("Mean Interest") +
scale_fill_brewer(palette = "Paired") + theme_minimal())
(Groups from https://paperswithcode.com/)
plotIterest.Cat(interest.m, "Computer Vision")
plotIterest.Cat(interest.m, "Audio")
plotIterest.Cat(interest.m, "Graphs")
plotIterest.Cat(interest.m, "Natural Language Processing")
plotIterest.Cat(interest.m, "Playing Games")
plotIterest.Cat(interest.m, "Miscellaneous")
plotIterest.Cat(interest.m, "Medical")
plotIterest.Cat(interest.m, "Methodology")
plotIterest.Cat(interest.m, "Speech")
plotIterest.Cat(interest.m, "Reasoning")
plotIterest.Cat(interest.m, "Time Series")
plotIterest.Cat(interest.m, "Computer Code")